Bitmap Filter: Speeding up Exact Set Similarity Joins with Bitwise Operations
نویسندگان
چکیده
The Exact Set Similarity Join problem aims to find all similar sets between two collections of sets, with respect to a threshold and a similarity function such as overlap, Jaccard, dice or cosine. The näıve approach verifies all pairs of sets and it is often considered impractical due the high number of combinations. So, Exact Set Similarity Join algorithms are usually based on the Filter-Verification Framework, that applies a series of filters to reduce the number of verified pairs. This paper presents a new filtering technique called Bitmap Filter, which is able to accelerate state-of-the-art algorithms for the exact Set Similarity Join problem. The Bitmap Filter uses hash functions to create bitmaps of fixed b bits, representing characteristics of the sets. Then, it applies bitwise operations (such as xor and population count) on the bitmaps in order to infer a similarity upper bound for each pair of sets. If the upper bound is below a given similarity threshold, the pair of sets is pruned. The Bitmap Filter benefits from the fact that bitwise operations are efficiently implemented by many modern general-purpose processors and it was easily applied to four state-of-the-art algorithms implemented in CPU: AllPairs, PPJoin, AdaptJoin and GroupJoin. Furthermore, we propose a Graphic Processor Unit (GPU) algorithm based on the näıve approach but using the Bitmap Filter to speedup the computation. The experiments considered 9 collections containing from 100 thousands up to 10 million sets and the joins were made using Jaccard thresholds from 0.50 to 0.95. The Bitmap Filter was able to improve 90% of the experiments in CPU, with speedups of up to 4.50× and 1.43× on average. Using the GPU algorithm, the experiments were able to speedup the original CPU algorithms by up to 577× using an Nvidia Geforce GTX 980 Ti.
منابع مشابه
Buddy-RAM: Improving the Performance and Efficiency of Bulk Bitwise Operations Using DRAM
Bitwise operations are an important component of modern day programming. Many widely-used data structures (e.g., bitmap indices in databases) rely on fast bitwise operations on large bit vectors to achieve high performance. Unfortunately, in existing systems, regardless of the underlying architecture (e.g., CPU, GPU, FPGA), the throughput of such bulk bitwise operations is limited by the availa...
متن کاملEfficient representation of integer sets
We present a solution that uses bitmaps and bitwise operators to represent nonnegative integer sets and implement some common set theoretical operations. We also show how it is possible to extend this representation to set partitions.
متن کاملFast Set Intersection through Run-Time Bitmap Construction over PForDelta-Compressed Indexes
Set intersection is a fundamental operation for evaluating conjunctive queries in the context of scientific data analysis. The state-of-the-art approach in performing set intersection, compressed bitmap indexing, achieves high computational efficiency because of cheap bitwise operations; however, overall efficiency is often nullified by the HPC I/O bottleneck, because compressed bitmap indexes ...
متن کاملCONCISE: Compressed 'n' Composable Integer Set
Bit arrays, or bitmaps, are used to significantly speed up set operations in several areas, such as data warehousing, information retrieval, and data mining, to cite a few. However, bitmaps usually use a large storage space, thus requiring compression. Nevertheless, there is a space-time tradeoff among compression schemes. The Word Aligned Hybrid (WAH) bitmap compression trades some space to al...
متن کاملA Hierarchical Bitmap Indexing Method for Similarity Search in High-Dimensional Multimedia Databases
This paper proposes an efficient indexing mechanism for similarity search in highdimensional multimedia database that quickly filter-outs the irrelevant objects using a novel indexing structure, called HBI (Hierarchical Bitmap Index). In this bitmap index, the feature (or attribute) value of object at each dimension is represented with a set of two bits each of which indicates whether it is rel...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1711.07295 شماره
صفحات -
تاریخ انتشار 2017